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1  Introduction 

This  is  the  Cognitive  Task  Analysis  (CTA)  report  (CDRL  Data  Item  #A006)  for  EAITS,  an  extensible  and 
adaptive  STEM  Learning  System  that  BBN  is  developing  under  the  ONR  STEM  Grand  Challenge  (GC) 
Program,  Contract  #  N00014-12-C-0535. 

EAITS  Phase  I  CTA  (i.e.,  prior  to  human  subjects  research  authorization)  focuses  on  identifying 
aspects  of  CTA  suitable  for  the  overall  system  concept  (Ref:  System  Concept/Specification  Report  #A003) 
and  planning  for  the  Phase  II  CTA.  During  Phase  II,  we  plan  to  conduct  the  CTA  in  two  stages.  First,  we 
will  focus  on  the  primary  learning  activity  employed  by  our  system  (problem-solving)  in  the  context  of 
in-domains  problems.  During  this  stage,  the  choice  of  learning  environments  will  be  set  aside  to  study 
the  fundamental  cognitive  processes  a  learner  applies  while  solving  SAT  Physics  problems.  Insights  from 
this  stage  will  be  used  to  improve  the  EAITS  learning  environment  being  developed.  The  second  stage  of 
CTA  will  focus  on  studying  problem  solving  within  our  learning  environment.  Observations  at  this  stage 
will  help  us  determine  how  well  our  learning  environment  aligns  with  underlying  cognitive  processes. 
Furthermore,  the  second  stage  CTA  will  overlap  with  preliminary  usability  testing  of  our  system. 

Section  2  describes  a  preliminary  model  of  the  problem  solving  learning  activity.  Section  3  describes 
our  plans  for  applying  an  iterative  CTA  procedure  for  each  of  the  two  stages  described  above. 

2  Preliminary  Model 


Figure  1.  Task  Diagram  of  SAT  Physics  Problem-Solving  Activity 
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We  intend  to  use  Cognitive  Task  Analysis  (CTA)  primarily  as  a  method  for  informing  and  improving 
the  design  of  our  learning  system.  We  have  adopted  a  simplified  approach  to  CTA  to  enable  us  to 
efficiently  iterate  through  multiple  cycles  of  design,  development,  and  (formal  as  well  as  informal)  user¬ 
testing.  In  this  approach,  we  start  from  a  preliminary  model  of  the  SAT  Physics  problem-solving  learning 
activity  shown  in  Figure  1. 

Analyze  problem:  The  problem  solving  activity  starts  with  an  analysis  of  the  problem.  The  learner 
identifies  the  objective  of  the  problem  and  enumerates  potential  approaches  to  solve  the  problem. 
Unlike  problem  solving  in  a  design  setting,  typically  there  are  only  a  small  number  of  applicable 
approaches.  In  a  time  constrained  SAT  Physics  problem  solving  setting,  it  is  crucial  that  the  learner 
quickly  selects  an  approach  and  proceed  to  the  next  step. 

Decompose  problem  into  sub-steps:  The  learner  identifies  the  sequence  of  sub-steps  involved  in 
applying  the  approach  to  solving  the  problem.  Typically,  this  is  performed  in  an  incremental  manner 
where  not  all  sub-steps  are  enumerated  in  advance,  but  instead  are  determined  as  the  learner  proceeds 
further.  We  believe  this  is  one  of  the  key  cognitive  process  that  a  problem-solving  based  learning  system 
should  support. 

Apply  skills/concepts  to  solve  sub-step:  The  learner  recalls  previously  mastered  skills  and  applies 
them  to  making  progress  on  the  sub-step  at  hand.  At  this  point,  the  learner  may  not  have  mastered  the 
required  skills  or  may  make  an  error  (slip)  in  the  application  of  the  skill.  Both  of  these  possibilities  must 
be  readily  supported  within  the  learning  environment  by  making  relevant  instructional  content  (such  as 
dialogs,  videos,  etc.)  available  to  the  learner  as  well  as  providing  timely  assessment/feedback  in  case  of 
a  slip. 

Compose  solution:  After  completing  all  the  sub-steps  involved  in  the  chosen  approach,  the  learner 
composes  the  solutions  to  each  sub-step  into  a  singular  solution  to  the  problem.  As  in  the  case  of 
analysis  of  the  problem,  this  step  could  be  very  demanding  in  the  generalized  problem  solving  activity. 
However,  due  to  the  nature  of  SAT  Physics  problems,  we  believe  this  step  usually  has  minimal  cognitive 
load  associated  with  it  in  our  application. 

Identify  Errors:  Within  time  bounds,  the  learner  may  consider  alternative  approaches  to  solve  the 
problem  that  may  rectify  errors  in  the  application  of  the  approaches  pursued  so  far.  This  step  in  the 
problem  solving  activity  allows  the  learner  to  reflect  on  the  approach  pursued  which  could  be  crucial  for 
the  learning  process. 

Note  that  each  process  shown  in  Figure  1  is  supported  by  different  components  in  our  current 
system  design.  For  example,  decomposition  of  a  problem  is  supported  by  making  on-demand  problem 
solving  help  available  to  the  learner.  Similarly,  system  capabilities  such  as  automated  assessment,  access 
to  integrated  multimedia  content  and  system  initiated  reflective  dialogs  support  other  processes 
depicted  in  the  task  diagram. 

As  we  move  forward  with  the  validation  and  refinement  of  this  CTA  (discussed  in  the  next  section), 
we  will  also  refine  our  system  design  and  implementation.  This  is  central  to  our  iterative  approach 
where  our  preliminary  CTA  informs  our  preliminary  system  design. 
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3  Plan  for  conducting  Cognitive  Task  Analysis 

As  discussed  in  Section  1,  our  next  steps  for  CTA  will  be  divided  into  two  stages.  In  the  first  stage, 
we  will  validate  the  preliminary  model  of  problem-solving  learning  activity  described  in  Section  2  using 
Think  Aloud  methods  in  the  context  of  a  SAT  Physics  problem-solving  task.  A  small  set  of  subjects  will  be 
asked  to  solve  3-5  SAT  Physics  problems  (from  the  topics  of  Electromagnetism  &  DC  circuits)  on  paper. 
The  subjects  will  be  asked  to  describe  their  thought  process  through  the  task.  Furthermore,  a  human 
tutor  (subject  matter  expert)  will  be  made  available  to  answer  conceptual  questions  as  well  as  provide 
hints  and  feedback  when  the  subject  is  unable  to  proceed  further  while  solving  the  problem.  We  will 
make  observations  about  the  time  spent  as  well  as  need  for  help  along  each  of  the  process  depicted  in 
Figure  1.  Also,  any  additional  cognitive  process  not  depicted  in  our  preliminary  model  will  be  identified 
and  incorporated  into  the  CTA  at  this  stage.  The  model  update  and  validation  cycle  may  be  iterated  if 
necessary. 

The  second  stage  of  CTA  will  focus  on  validation  of  the  refined  CTA  from  Stage  1  in  the  context  of 
the  problem-solving  using  our  system  implementation  at  the  time.  Similar  to  the  first  stage,  a  small  set 
of  subjects  will  solve  3-5  SAT  Physics  problems  using  our  learning  environment.  The  learner  will  be  able 
to  use  the  different  features  of  the  system  including  multiple  forms  of  support,  access  to  instructional 
content  and  automated  assessment  while  solving  problems  in  the  learning  environment.  CTA  validation 
and  refinement  at  this  stage  will  primarily  focus  on  identifying  the  differences  between  problem-solving 
on  paper  vs.  in  an  online  learning  environment  in  terms  of  metrics  such  as  time  spent  and  help  needed. 

The  refined  CTA  obtained  after  these  two  stages  will  serve  as  the  reference  model  for  making  final 
design  modification  to  our  learning  system  before  proceeding  to  first  formal  evaluation  of  the  system 
during  Phase  II  of  this  effort. 
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